-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update parser tests to validate token ranges #12019
Conversation
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a good assertion to add!
The title of this PR is a bit confusing because it is identical to another PR of yours. Maybe rename one of them?
e7cb731
to
a92f5e0
Compare
## Summary This PR updates the unterminated string error range to not include the final newline character. This is a follow-up to #12016 and required for #12019 This is not done for when the unterminated string goes till the end of file (not a newline character). The unterminated f-string range is correct. ### Why is this required for #12019 ? Because otherwise the token ranges will overlap. For example: ```py f"{" f"{foo!r" ``` Here, the re-lexing logic recovers from an unterminated f-string and thus emitting a `Newline` token for the one at the end of the first line. But, currently the `Unknown` and the `Newline` token would overlap because the `Unknown` token (unterminated string literal) range would include the newline character. ## Test Plan Update and validate the snapshot.
b7a76b2
to
1a01c00
Compare
Summary
This PR updates the parser test infrastructure to validate the token ranges.
From the code documentation:
Follow-up from #12016 and #12017
resolves: #11938
Test Plan
Make sure that there are no failures.